Repository navigation
FEAT: Save techniques (part 14) - #4066
Open
varunj-msft wants to merge 531 commits into
Open
varunj-msft wants to merge 531 commits into
varunj-msft wants to merge 531 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A tag-related 409 conflict is stored only against the technique name, so correcting Tags leaves Save disabled and forces an unnecessary rename before retrying.
1 open issue
What changed in this PR
Extends registry persistence to attack techniques, building on #4052’s converter-saving work. Techniques can survive backend restarts while retaining the existing temporary-creation and per-run editing behavior.
Changes:
- Adds saved technique requests, recipe metadata, restore integration, and dependency/conflict checks.
- Makes the technique editor save by default, with Temporary fallback messaging and registry origin badges.
- Includes stacked persistence safeguards, target ownership display, documentation, and isolated test storage.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging safeguards. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement refusal. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe storage configuration validation. |
| tests/unit/registry/test_instance_registry.py | Tests origin metadata and replacement safeguards. |
| tests/unit/registry/test_converter_registry.py | Tests saved-converter conflict recording. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique registration safeguards. |
| tests/unit/backend/test_technique_edit_service.py | Updates creation calls to keyword arguments. |
| tests/unit/backend/test_scorer_routes.py | Tests scorer persistence and error responses. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch conflict reporting. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage responses and logs. |
| tests/unit/backend/test_saved_entries.py | Tests temporary dependency rejection. |
| tests/unit/backend/test_mappers.py | Tests target origin and ownership mapping. |
| tests/unit/backend/test_initializer_service.py | Tests unreadable initializer storage handling. |
| tests/unit/backend/test_auth_middleware.py | Tests authenticated object-ID lookup. |
| tests/unit/backend/test_analytics_benchmark.py | Tests benchmark recipe storage isolation. |
| tests/unit/backend/conftest.py | Isolates saved entries during backend tests. |
| tests/end_to_end/conftest.py | Gives launched backends temporary recipe storage. |
| test_support/analytics_benchmark.py | Configures isolated benchmark recipe storage. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry collisions during tool setup. |
| pyrit/setup/initializers/targets.py | Protects saved targets during automatic grouping. |
| pyrit/setup/initializers/scorers.py | Prevents saved scorers replacing configured defaults. |
| pyrit/setup/configuration_loader.py | Adds recipe storage source configuration. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares local classifier persistence restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares audio scorer persistence restrictions. |
| pyrit/score/scorer.py | Adds scorer saveability metadata contracts. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares credential fallback behavior. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares audio scorer persistence restrictions. |
| pyrit/registry/instance_registry.py | Tracks origins, owners, and saved-entry conflicts. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/attack_technique_registry.py | Adds conflict prechecks and saved-technique safeguards. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares account-dependent persistence restriction. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/openai/openai_target.py | Declares credential, header, and destination policies. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback behavior. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model persistence restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Disallows saving credential-bearing raw requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback behavior. |
| pyrit/prompt_target/github_copilot_target.py | Declares token fallback behavior. |
| pyrit/prompt_target/common/prompt_target.py | Defines target persistence policy metadata. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and endpoint policies. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares token and container destination policies. |
| pyrit/prompt_target/a2a_target.py | Declares endpoint and agent-card destination policies. |
| pyrit/models/catalog/target.py | Adds target origin and ownership fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin field. |
| pyrit/models/catalog/registry_recipe.py | Defines saved recipes and restore reports. |
| pyrit/models/catalog/__init__.py | Exports recipe and restore models lazily. |
| pyrit/converter/converter.py | Defines converter credential fallback metadata. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Declares Speech credential fallback behavior. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Declares Speech credential fallback behavior. |
| pyrit/backend/services/technique_service.py | Saves techniques and exposes their origins. |
| pyrit/backend/services/scorer_service.py | Saves scorers and exposes their origins. |
| pyrit/backend/services/scenario_suite_execution_service.py | Surfaces saved-entry launch conflict messages. |
| pyrit/backend/services/scenario_run_service.py | Detects saved-entry conflicts in launch scripts. |
| pyrit/backend/services/runtime_lifecycle.py | Restores saved entries and reports failures. |
| pyrit/backend/routes/techniques.py | Adds persistence context and typed save errors. |
| pyrit/backend/routes/targets.py | Adds persistence context and caller-specific ownership. |
| pyrit/backend/routes/scorers.py | Adds persistence context and save errors. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage diagnostics and handles listing failures. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage diagnostics and handles listing failures. |
| pyrit/backend/routes/initializers.py | Handles local storage failures with safe diagnostics. |
| pyrit/backend/routes/converters.py | Adds save errors and saved-removal protection. |
| pyrit/backend/routes/common.py | Shares owner extraction and save-refusal responses. |
| pyrit/backend/README.md | Documents persistence APIs and origin semantics. |
| pyrit/backend/models/techniques.py | Adds persistence validation and origin metadata. |
| pyrit/backend/models/targets.py | Adds saveability metadata and persistence validation. |
| pyrit/backend/models/scorers.py | Adds scorer persistence validation. |
| pyrit/backend/models/converters.py | Adds saveability, origin, and persistence validation. |
| pyrit/backend/middleware/auth.py | Exposes the authenticated user’s object ID. |
| pyrit/backend/mappers/technique_mappers.py | Includes technique origin in responses. |
| pyrit/backend/mappers/target_mappers.py | Includes target origin and ownership display fields. |
| pyrit/backend/mappers/converter_mappers.py | Includes converter origin in responses. |
| frontend/src/utils/registryOrigin.ts | Defines shared origin labels. |
| frontend/src/utils/registryName.ts | Generates valid, available registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds persistence, origin, and ownership types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and ownership test data. |
| frontend/src/services/errors.ts | Recognizes typed save-refusal errors. |
| frontend/src/services/errors.test.ts | Tests save-refusal recognition. |
| frontend/src/components/Techniques/TechniqueEditor.tsx | Saves by default and handles temporary dependencies. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates nested converter save interactions. |
| frontend/src/components/Registry/TechniqueRegistry.tsx | Displays technique origin badges. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Displays scorer origin badges. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests scorer origin labels. |
| frontend/src/components/Registry/OriginBadge.tsx | Provides a shared origin badge. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Labels origins and disables saved-converter removal. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests origins and saved-removal restrictions. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Clarifies that converter copies remain Temporary. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests Temporary-copy messaging. |
| frontend/src/components/Config/TargetTable.tsx | Shows origins, owners, and an ownership filter. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests origin labels and ownership filtering. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles ownership captions and filter controls. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline backend recipe storage. |
| frontend/e2e/config.spec.ts | Tests target saving and ownership display. |
| frontend/e2e/_targets.ts | Adds origin and ownership mock fields. |
| doc/gui/0_gui.md | Explains saving, Temporary entries, and removal limits. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restore failures. |
| .pyrit_conf_example | Adds commented recipe storage configuration. |
Review effort: Balanced
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 23:56
# Conflicts: # pyrit/backend/middleware/error_handlers.py # pyrit/backend/routes/scenario_presets.py # pyrit/backend/routes/scenario_suites.py # pyrit/registry/file_document_storage.py
…y the other storage tests do
# Conflicts: # pyrit/backend/README.md
# Conflicts: # pyrit/backend/README.md
… a saveability answer once its request goes away The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
…s, and auth mode only
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and local refusal scorers. A saved entry still can't hold a typed key.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
This is one of the changes for saving Registry entries across restarts. It builds on #4052, which saves converters from the Registry. Until the earlier PRs merge, the diff here also shows their changes.
A technique made with New technique, or through
POST /api/techniques, was gone after the backend restarted. It now saves by default and comes back at every start, like saved targets and converters.POST /api/techniquestakes"persist": true. A saved technique's name follows the saved-name rule (lowercase letters, digits and underscores, starting with a letter, up to 64 characters), or the request gets a 422. Its recipe is the create request without the name, plus who saved it./errors/not-saveable). Its attack type and attack parameters get the checks restore runs, so a type that can't be saved, a credential, or a Temporary entry named in the parameters is refused the same way. A name another technique has, or a tag that matches a technique's name or another spelling of a tag in use, is a 409 before anything is written, and storage that can't be written is a 503.GET /api/techniquesgives each technique anorigin:saved,configurationortemporary.privatein its recipe.What stays the same: a technique created without
persistis Temporary, as before. Settings changed for one run in the Scanner (POST /api/techniques/{name}/edit) are never saved. Startup techniques work as before; they're only labeled From configuration.Screenshots: New technique saves by default, the saved technique in the Techniques table, and a technique that uses a Temporary converter offered as Temporary with the reason.



Not covered:
registry_recipes_sourceand restart the backend.